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l\TY0-2878 
A  DUALITY"  THEOREM  FOR    CONVEX   PROGfiAHS 

1.  Introduction. 

In  a  recent  paper  [Ij,  the  duality  relationship  of 
linear  programming  was  extended  to  a  class  of  quadratic 
programs.   In  what  follows  this  concept  will  be  extended 
further  to  Include  convex  programs.  I.e.,  programs  to 
minimize  a  convex  function  subject  to  linear  constraints. 
The  arguments,  to  some  extent,  parallel  those  given  in  the 
quadratic  case.   The  notion  or  duality  for  this  class  of 
programs  has  also  been  discussed  previously  by  Dennis  [2]. 

2.  Notation. 

In  what  follows  matrix  notation  will  be  used.   Lower 
case  letters  x,y,...   will  denote  column  vectors  and 
capital  letters   A,C,...   will  be  matrices.   The  operator 
S/   is  the  gradient  operator  and  ^7  f (x)   is  a  vector 
composed  of  the  first  partial  derivatives  of  the  scalar 
function  f(x)  . 

Prime  denotes  transpose  so  that  x',y*,...   are  row 
vectors.   The  product  x'y   is  the  inner  product  of  the 
vectors  x   and  y.   A  vector  inequality  will  apply  to  each 
component  of  the  vector,  i.e.   x  >  0   implies  that  each 
component  of  x   is  non- negative. 
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3.   Convex  Functions. 

A   scalar  function,      f,      of  an  n-dimenslonal   vector, 
X,      Is   convex   If   linear   interpolation  does   not   underestimate 
the   value   of   the   function,    i.e., 

f(x^  +  k[x2-x^])      <     f(x^)   +  klfCx^)    -  f(x^)j 

for   0  <  k  <  1.   For  convex  functions  a  local  minirtiimi  is 
also  a  global  minimum. 

For  the  proofs  in  the  succeeding  sections  the  following 
lemma—'  regarding  convex  functions  is  required. 

Lemma.   If  f (x)   is  convex  then  for  any  vectors  y   end  x 

f(y)  -  f(x)  >   (y-x)'  V  f(x)  (3.1; 

Proof.   From  the  assumption  of  convexity 

f (x  +  k[y-xj)   <  f(x)  +  k[f(y)  -  f{x)J 
for  all  k  such  that   0  <  k  <  1,   Rearranging  terms 

k[f(y)  -  f(x)J   >  f(x  +  k[y-x])  -  f(x)  .     (3.2) 
From  the  differential  mean  value  theorem 

f(x  +  k[y-x])  -  f(x)   =  k(y-x)»  V  f(x+x^) 
where  x  =9  k(y-x)   for  some   9   such  that   0  <  9  <  1. 


TT 


The  author  is  indebted  to  C.  E.  Lemke  for  this  lemma, 
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Using  this  In  the  right  hand  member  of  (3»2) 

f(y)  -  f(x)  >   (y-x)'  V  i'(x  +  x^j 
Letting  k  approach  zero,  the  inequality  (3-1)  results. 


ij..   Duality  in  Linear  Programming. 

The  linear  programming  problem  may  be  posed  as  follows: 
To  minimize  the  linear  form  p'x   over  all  n-dimensional 
vectors  x   satisfying  the  constraints 

A  X  >  b  (i]..l) 

X  >   0  (i4..2) 

where   p   is  an  n  x  1   vector,   b   is  an  m  x  1  vector 
A   is  an  m  X  n  matrix. 

The  dual  problem  to  the  above  is  to  maximize  b'v   over 
all  m-dlmensional  vectors   v   satisfying 

A  V  <  p  (I1..3) 

V  >   0  (J+.l^) 

The  duality  theorem  [3]  states  that  if  a  solution  to 
either  problem  exists  and  is  finite,  then  a  solution  to 
the  other  problem  also  exists  and  Indeed 

Minimum  p'x  =  Maximum  b'v  (ij-.S) 

The  constraints  ([|..  1),  il^,2}    in  the  primal  problem  may  take 
on  other  forms  with  corresponding  changes  in  the  dual 
constraints  {l\-„3),    (^-.i|).   One  form  which  will  be  of  parti- 
cular interest  is  given  here.   Let  the  vector  x  be 
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partitioned   as       (x.jXp)      and   consider   the   problem   of 
inlnlmizing     P]_x,    +  P2^2      ^^^J®^*^   "^o 

A-jX-,    +  k^2^     >      ^  (I4-.6) 

x^      >      0  (Ij..?) 

where     p      and     A     have   been  similarly  partitioned.      Note 
that      Xp      is   not   required   to   be    non-negative.      These 
constraints    can  be   put    into   the   form   of    (l^.l)    and    (I4..2) 
by   the    substitution 

X2      =      z    -  y 

where      z  >   0    ,      y  >   0    .      All  variables     x,    ,    z   and  y     are 
now   non-negative. 

The    dual   problem   is   then  from    (It..3)    and    ([(..li) 

Maximize     b'v  (I|.«8) 


subject   to 


1 

*1 

V 

< 

Pi 

1 

A2 

V 

= 

P2 

V 

> 

0 

(1^..9) 

{I4-.IO) 

(i^-.ll) 


5.      A  Class    of   Convex   Programs   and   Its   Dual. 

Consider  the   following  class    of   non-linear  programming 
problems : 

Minimize:      0(x)      =     f(x)    +  p»x  (5.1) 

subject   to 
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Ax  >   b  {$'2) 

X  >   0  (5.3) 

where   f  (x)   Is  a  convex  scalar  function  of  the  vector  x 
and  possesses  continuous  first  partial  derivatives  with 
respect  to  each  component  of  x.   Moreover,  it  will  be 
assumed  that  given  a  vector   y,   the  vector  equation 
\7  f (x)  =  y  possesses  a  solution  x  which  is  one  differ- 
entiable  with  respect  to  y  .   The  vectors  p,b,  and  x  and 
the  matrix  A  are  the  same  as  in  Section  3»   Denote  this 
as  Problem  I, 

A  dual  problem  to  Problem  I  is 

Maximize   g(u,v)   =  f(u)  -  u»  V  f(u)  +  b»v  (S.lj-) 

subject  to 

a'v  -  V  f  (u)  <  p  (5.5) 

V  >   0  (5.6) 

Denote  this  as  Problem  II. 

Problems  I  and  II  are  related  through  the  theorem  in 
the  next  section. 

6.   A  Duality  Theorem  for  Convex  Programs. 

Theorem;   (i)   If  there  exists  a  vector  x   which 
o 

minimizes   0(x)   in  Problem  I,  then  there  also  exists 

vectors  u  =  x  ,  v  =  v   which  maximize   g(u,v)  in 
o        o  o   » 

Problem  II. 
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(ii)  Conversely,  if  u  ,v  are  vectors  which 
maximize  g(u,v)  in  Problem  II,  then  x  =  u^  minimizes 
0ix)      in  Problem   I.      In  either   case 

Mlnimiom     0{x)      =     Maximvim     g(u,v)    . 

Proof:      (i)      Let     x        be    a   solution  to   Problem  I,    then 

0{x^)     <     0U)  (6.1) 

Consider  the  following  linear  programming  problem 

Minimize  F(x)   =   -f  (x  )  +  x'  V  ^^^q^   "^  ^'^  ^^'^^ 

Ax  >  b  (6.3) 

X  >  0  (6.1^.) 

Call   this   Problem   I    ,      Note    that      x        is    a   feasible 

o 

t 

solution  to  Problem  I   in  the  sense  that  it  satisfies 

the  constraints  (6.3)  and  (6.1;.).   Suppose  there  exists 
another  feasible  solution  x   to  Problem  I  with  the 
property  that 

F    (x'^)      <     F(x    )    , 


(x^'-x^)'    [V  f(x^)+p]      <     0  (6.5) 


o 
It   is   easily  verified  that 

Xt      =     X     +  k(x'-x    )    ,  0  <  k  <  1 

10  o  —       — 

is  also  a  feasible  solution  to  both  Problem  I  and 

Problem  I  ,   Consider 
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0{x^)    -  0U^)      =     [f(x^)+p'x^]    -    [f(x^)+p'x^] 
From  the  mean  value    theorem 

0U^)  -  0U^)    =     (x;j^-x^}'   [V  f(x^-e[x^-x^"^}+pj   ,  o<e<i 

=     k(x''"'-x^)'    LV  f  (Xo-©k[x'"'-x^^)+pJ 

=    k[{x''-x^)\\/  f(x^-ek[x''-x^j)-V  ^^^Ji 

+    (x''-x^)'    [V  f(x^)    +  pj? 

The    second  term   in  the   braces   is   negative  by  vlrute    of   the 
assumption   (6.5) •      Moreover,    this    term  is   independent    of 
k.      However,    since     XJ  f(x)      is   continuous,    for      k 
sufficiently  small   the   first   term   in  the  braces  may  be 
made   arbitrarily   small.      In  particular      k     may  be    chosen 
so   that 

(x''''-x^)'[Vr(xQ-ek[x''-x^])-Vf(x^)]|  <  |{x'''-x^}'[Vf(x^''+pJ 

It  follows  then  from  the  inequality  (6.5)  that 

0(x,  )  -  0{x)      <   0 
1       o 

But  this  is  in  contradiction  to(6..1).   Therefore,   x^ 

t 

must  be  the  minimizing  solution  of  Problem  I  . 

Prom  the  duality  theorem  of  linear  programming  a 

r 

solution  also  exists  to  the  dual  to  Problem  I  *   This  dual 
linear  problem  is  from  (J4..3)  and  {h\..l\.)» 

Maximize   G(v)   =   -  f  (x^)  +  b'v       (6.6) 

a'v  <  f(x  )  +  p  (6.7) 

—  o 

V  >  0  (6,8) 
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Call  this  Problem  II  .   Denote  the  vector   v  which 
majcimlzes  Problem  II  by  v^.   Then  the  duality  theorem  of 
linear  programming,  (l+.S),  requires  that 


x'  V  f(x  )  +  x'p  =  b'v^  .       (6.9) 
o       o      o         o 


Notice  now  that  u  =  x  ,  v  =  v^  comprises  a  feasible 
solution  to  Problem  IIo  To  show  that  it  also  provides 
a  maximal  solution  consider 

g(x  ,v  )  -  g(u,v)   =  f(x  }'xl\7f{x)+h^v-f{u)+u^\7fiu)Jo'v 
'-'      o   o  ooo      o 

Prom  (6.9) 

g(x  ,v  )  -  g(u,v)   =   f(x^)  -  f (u)+x'p+u'V^(u)-b'v 
"-•oo  o  o 

and  from  the   lemma 

g(x^,v^)-g(u,v)      >      (x^-u)'Vf(u)+x^p+u'Vi'(u)-b'v 

=     x'    [V  f(u)    +  p]    -  b'v        (6.10) 
o 

Now   from    (5.2)    and    (5.6) 

I  ! 

-b'v     >      -X     A      V 
—  o 

and  from    {$»3)    and    i^'S) 

x^    [V  f  (u)    +  p]     >      x^  a'v 

Substituting  these    last    two   inequalities    in    (6.10) 

g(x^,v^)    -   g(u,v)      >      x^Av-x^Av      =      0 

Thus   (x  ,v  )  is  a  maximizing  solution  for  Problem  II. 
o   o 

Finally  from  iS.k) ,    (6.9)  and  iS.D 

g(x^,v^)   =  f  (xjVVf(x^)+b.v^  =  f(xj^;^  =  0U^) 


-  11 


which  verifies   the   equality   of  the    objective   functions. 

(li)    Let       (u   ,v   ,j    )      be    a  maximizing   solution  of 
Problem  II   rephrased   as 

Maximize      \|'(u,v,y)      =     f(uj  -  u«y  +  b'v               (6.11) 

Av-y<p  (6.12) 

y     =     V  f(uJ  (6.13) 

V     >      0  (6.1[^.) 

Thus 

t(u^,v^,y^)      >      \l'(u,v,y)  (6.15) 

for   all     u,v,y      satisfying   the    constraints. 
Consider   the   related  linear  problem 

Maximize      G(v,y)      =     ^^'^^^    "  ^^7  '^  b'v 

A  V   -  y     <     p 

V     >      0 

Denote  this  as  Problem  II". 

Notice  that   v=v,   Y=y   is  a  feasible  solution 

o  o 

to  Problem   II    .      Suppose    there   exists      v' ,y'       satisfying 
(6.12)    and    (6.11^)    such   that 


i.e.. 


G(v*,y*j     >     G(v^,y^)    , 


'  /     -»  V       .      ,     .  /      ■» 


-u    (y  -y    )    +  b'(v*-v    )     >     0  (6.16) 

o  *^  o  o 


Define   vectors      v,  ,y,      to  be 
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Vt   =  V  +  k(v*-v  )  (6„17) 

1      o         o 

71  =  7o  +  k(7*-7^J  (6.18) 

for   0  <  k  <  1.   Then  v^  ,y-,   is  also  a  feasible  solution 
to  Problem  II  .   Now  by  assumption  (Section  5)  there  exists 
a  vector  u,   such  that 

y^  =  V  f  (u-^)  (6.19) 

and  thus  i^-i»v,,y,   is  a  feasible  solution  to  Problem  IIo 
Consider 

^(U]_,v^,y^)  -  t(u^,v^,y^)   =  f  (u^)-u^y^+b 'v^-f  (u^) 

+  u  y   -  b  '  V 
o'^o      o 

Prom   the  mean  value    theorem 
^(u-L,v^,7]_)    "   ^^%»^o*^o^      ^      ^^l'^-*     V  f(u^-e[u^-u^j) 

-  u^7i+u^7o+b ' ( v^-v^ )      0<G<1 

and  using   equation   (6«19) 

^(ui,v^,y^)    -   H'(u^,v^,y^)      =      (u^^-u^) '  [ Vf  (u3_-©[u^-u^ J ) 

-)        ,  (6.20) 

-Vf(u^)J-u^(7i-7^)-^b.(v^-v^) 

Since     u     possesses   first  derivatives  with  respect   to      y, 
where     y  =  XJ  f(u),      it   follows  from  the  mean  value    theorem 
that 

^^1"%^      ^      ^^I'^o'*      ^^7o-p[7i-7o])    ,  0<Pll 

where     A(y)      is    a  matrix  whose      i,j      element    is   the   partial 
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derivative    of  the    jth   component    of     u     with  respect   to  the 
ith  component   of     y.      Using  this   and  equations    (6.17)    and 
(6.18),    equation    (6.20)   becomes 

^(u-L,v^,y^)    -   ^(u^,v^,y^)   = 

k[(y*-yo^'  A(y^-pkLy-L-y^])[V  f  (u^-ek[u'-u^j  )-Vf  (u^  j 

-  ^o^y%o^   "^  b»(v*-v^)J  (6.21) 

Now  if   k  is  sufficiently  small,  the  following  inequality 
can  be  satisfied. 

|(y*.7^)'  A(y^-pk[y*-yJ)[Vf(u-L-ek[u''-uJ)  -  ^f  [u^)'j  \ 

<    I-  u;(y%^)  +  b«(v*-v^)| 

since  ^f  (u)   is  continuous  and  the  term  on  the  right  is 
independent  of   k. 

Prom  the  assumption  (6.16),  the  term  inside  the 
absolute  value  signs  on  the  right  of  the  above  inequality 
is  positive.   Thus  the  term  in  square  brackets  in  equation 
(6.21)  may  be  made  positive  by  a  sufficiently  small  choice 
of  k.   This  implies  that 

^(u^,v^,y^)  -  n'(u^,VQ,y^)  >   0 

and  contradicts  the  hypothesis  (6.15)  that   u  ,v  ,y   is  a 

•'■^  o   o   o 

ma:xiraizing  solution  of  Problem  II.   It  follows  that   v  ,y 
maximizes  Problem  II  . 
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Problem  II   is  a  linear  problem  of  the  second  type 
mentioned  in  Section  l\.,    equations  {[j..6j  and  (I4..7),  and 
its  dual  problem  according  to  (1^..  d)  to  (14..  11}  is 

Minimize   F(x)   =  f(u^)  +  P'x         (6.22) 

Ax  >  b  (6.23) 

X  =  u  (6,214.) 

o 

X  >  0  (6.25) 

This   will  be   referred  to    as   Problem   I    .      The    duality   theorem 
of   linear  programming   assvires   the   existence    of    a  solution  to 
this  problem.      By  virtue    of    (6.2I4.)      u       must  be   that    solution. 
The    duality  theorem,    (i;.5),    then  provides   that 

Maximum     G(v,y)      =     Minimum     F(x) 


or 


-  u  Y     +  b'v        =     p'u  (6.26) 

o''  o  o  -^      o 

Now  u  ,   since  it  satisfies  (6.23)  and  (6.25),  is  a 

feasible  solution  to  Problem  I.   Let   x  be  any  other 

feasible  solution  to  Problem  I  and  consider 

0U)    -   0(u^)   =  f(x)  -  f(u^)  +  p'(x-u^) 

Prom  the    lemma   again 

0{x)    -  0{u^)      >       (x-u^)'[Vf(u^)+pJ  (6.27) 

Now   from    (6.13)    and    (6.26) 

-    u'      [V    f(u     )+PJ        =        -    l^'(y    +P^        =        -    b'^n 


O    "  O 
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and   from    {S»2)    and    (5*6) 


t 

-  b  'v        >      -  x'    A     V 
o     —  o 


so 


-  u^   [V  f(%)    +  PJ     >      -  X'   a'    v^  (6.28) 

From    (5,3)    and    (5.5) 

x»    [V  f(^io)   +  P]     >     X'   a'   v^  (6.29) 

Substituting    (6.28)    and    (6.29)    into   the   right   hand  member 
of    (6.27)    it   follows   that 

0{x)    -   0in    )      >      x'A   V      -   x'A   v^      =      0 
o        —  o  o 

Thus   u   minimizes  Problem  I. 
o 

Prom  (5.1),  (6.26)  and  (5.1;) 
^(u^)   =   f(u^)+p.u^  =  f(u^)V  Vf(Uo)+b'v^=  g(u^,v^) 

7.   Remarks. 

It  should  be  noted  that  the  rather  strong  assumption 
(Section  5)  that  y  =  \^fi.x)      has  a  dif ferentiable  solution 
X  for  all  y   is  necessary  only  for  the  proof  of  part  (il) 
of  the  theorem.   That  is  to  say,  the  existence  of  a  solution 
X   to  Problem  I  implies  the  existence  of  a  solution  x^,v^ 
to  Problem  II  even  when  this  assumption  is  not  valid. 

The  significance  of  the  assumption  may  be  seen  from 
the  quadratic  case  where  f(x)  =  l/2  (x'Cx)  .  The  above 
assumption  then  requires  that   C  be  positive,  definite. 
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The   duality   theorem  for   quadratic  programs    [1]   requires 
only  that      C     be   positive,    semi-definite    and  the   duality 
theorem  for    convex  programs   presented  here    is,    in  that 
sense,    less   general   than  the   corresponding  quadratic 
theorem. 

The   proof   of   Section  6  provides   a  method  for   comput- 
ing the   dual  variables     v        since    the   primal   variables     x 

have   been  obtained.      Indeed,      v        is    the    solution  of  the 

'    o 

following  linear  prograinming  problem 

Maximize  b'v 

subject  to 

a'v  1  P  +  V  ^(x  ^ 
V  >   0 

Finally  the  constraints  of  the  primal  problem 
(Problem  I)  may  be  formulated  in  other  ways  with  corres- 
ponding changes  in  the  dual  constraints.   Some  of  these 
are  tabulated  below.   In  all  cases  the  functionals  to  be 
optimized  are  the  same  as  those  in  Section  5« 

Problem  I  Type  Problem  II 

Ax  >   b  I         a'v  -  V  f(u)   <   p 

X  >   0  V  >   0 


Ax     >     b 

II 

A   V    -  V  f (u)      =      p 
V     >      0 

Ax     =     b 

III 

a'v    -  V  f(u)      <      p 

X      >      0 

Ax  =  b  IV       A  V  -  V  f  (u)   =  p 
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8.      Example 

As   a   simple   example    of   the   theorem   of   Section  6, 
consider   the   problem   of  minimizing 

f(x,,X2)      =      -  log  X-,    -   log  Xp 

subject   to 

-  X,    -  Xp     >      -2 

X,      >      0 

x^     >      0 

The    admissible    values   of     x, jXp      lie    in  the   triangle 
OAB      of  PigTju?e   I.      The    contours    of    constant   values    of     f 
are   also    shown  with     f      decreasing   as   the    contours   recede 
from  the    origin. 

The    optimal   value   is      f  =  0      for     x,    =  Xp  =  1. 

The    dual   problem   is 

Maximize   g(u-,,Up,v)   =   -  log  u,-log  Up+Z   -   2v 
where 


- 

v 

+ 

1 

^1 

< 

0 

- 

V 

+ 

1 

^2 

< 

0 

V  >   0 

From  the  symmetry  of  the  dual  it  is  clear  that  when  g  is 
a  maximum,   u-,  =  Up.   The  admissible  values  of  u-,   and  v 
are  those  above  the  hyperbola  CD  in  Figure  2,   The 
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Flgiore   1 
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Figure   1 
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contours   of   constant      g     are   shown  as   dotted   curves  with 
g      increasing   as   the    contoiirs   approach  the    origin.      The 
raaximimi  value    of     g      is    associated  with   the    contoiu?  which 
is   tangent   to   the   hyperbola     CD     at     u,    =  Up   =  v  =   1. 
For   these   values   of    the  variables      g(l,l,l)    =   0    . 
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